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An attempt is aade to deternine whether a procedure 
requiring each leaber of a rating group to rank a subset of fifteen 
goals selected at j;andoBi froa one hundred and six goals vould produce 
result^ that were, in effect, equivalent to having eacL aeiber of a 
rating group rate each of the one hundred and six goaI?>. The ratings 
of the one hundred and six goals by fifty^three Califcxnia elementary 
school principals obtained during a field testing were us^d to 
provide t^e standard Against which the results of the subset ranking 
procedures vas to be compared since the sampling of each group was 
largely incidental and no provision vas made for equating or 
randomizing administrators between the two rating groups, differences 
observed might be influenced by extraneous variables. The subset 
ranking procedure (SSK) yields results that are equivalent to those 
obtained with complete rating or ranking procedures. However, it 
seems inappropriate at this time to either recommend or reject the 
use of the SRK or any other subset procedure as! a substitute for 
complete rating/ranking procedures until the potential problem of 
rater dissatisfaction can be investigated < (Author/BB) 
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The Collective Viewpoints Approach to goal selection described in the CSE 
Elementary School Evaljation KIT: Needs Assessment (Hoepfner, et al . , 1972) 
provides a means for systematically determining v^at various group (such as 
principals, teachers, parents, and school boards) consider to be the relative 
priorities among the set 106 elementary school goals described in the KIT 
(see Appendix). Basically, the Collective Viewpoints approach involves 
(1) having each member of the rating-group rate each of the 106 goals on a 
fiye-point scale of importance; and then (2) rank -ordering the goals accord- 
ing to their average ratings. An optional procedure for pinpointing the most 
important goals requires that the rating-group rank the top-priority goals 
identified in number (1), and then proceed with number (2) based on average 
rankings. 

The purpose of the present study was to determine whether a procedure 
requiring each member of a rating-group to '*rank" a subset of only 15 goals 
selected at random from the 106 included in the KIT would produce results 
that were,^in effect, equivalent to having each member of a rating-group 
rate each oj the 106 goals. 

' The equival^ce of the results of the ''subset- ranking procedure" and ^ 
"complete -ranking procedure" would provide a basis for considering the^less 
resource -demanding subset -ranking procedure (i.e., fewer demands in terms of 
rater time arid effort^ rater ability to. discriminate goal priorities, -tally- 
ing time, and number of required goal cards) as an alternate or substitute 
for a complete rating or ranking procedure. 



Ratings of the 10b goals by fifty-three C^ai fonu.'. e omentary school 
principals obtained during the f iel J-tosting of the Needs Assessment KIT 
(June - July 1970) provided the standard against which the results of the 
subset ranking procedure was to he compared. The -tandard ratings were ob- 
tained in the following manner: 

Each principal participating in the field- test was iastructed to 
rate each of the 106 goals on a five-point scale: 

1, Unimportant, Irrelevant 

2. Marginal Importance 
2. Average Importance 
4* Moderate Importance 
5. Most Important 

Two weeks'^ater they were asked to rate each goal again and then 
to calculate their average rating for each goal. Each pr^ icipal 
then rank-ordered all 106 ave:ages, CSE calculated the principals' 
mean rating for each goal and the Principals' mean ranking for each 
goal. The mean ratings and the mean rankings were then rank-ordered; 
the ratings supplying a '^Complete Rating Procedure" (CRT) ranking, 
and the ranl;ings supplying a **Canplete Ranking Procedure" (CRK) rank- 
ing, , 

The results of the subset -ranking procedure were obtained in the folio 



Complete decks of 106 goal cards were randomly sorted into 15-card 
and 16-card subdecks (each 106 card deck yielded six 15-card sub- 
.decks and one 16-card subdeck). Between November 1970 and Janup^ 
1971, 74 California school administrators (^principals, superintendents, 
and assistant superintendents) attending workshops around the state 
were asked to rank-order the goals in the subdecks according to their 
importance, CSE then calculated a mean rank value for each goal and 
.rank-ordered the values to^yield "Subset Ranking Procedure'^ (SRK) 
results. 



It should be made clear that i>^ile botl groups 6f raters were Caliromia 
school administrators, the two might well differ in many important w; ^, 
Since the sampling of each group was largely incidental and no provision was 
made for equating or randomizing administrators between the two rating groups 



manner: 




differences observed might well be influenced by all the extraneous Variables 
that confound the worst of the quasi-experimental studies. With this kept in 
mind, the reader can interpret the following indexes of similarity of ratings 
as very low estimates of the similarity that would have been demonstrated if 
experiinental sampling techniques had been rigorously adhered to. 

RESULTS 

The results of employing the SRK to determine goal priorities are re- 
port ! in Table 1. The goals in the table have been arranged according to 
the means of their rank values. 

Spearman Rank Correlation Coefficients (r^) and Kendall Tau Correlation 
Coefficients (T) were calculated to obtain indexes of the extent to which the 
results obtained by employing the three procedures were related. All coeffi- 
cients in Table 2 are significant at the .001 level. The null hypothesis that 
the observed values of r^ and T differ^ f ran zero only by chance and that the 
goal prioritizing procedures do not yield results that are related must be 
rejected. ''^ 

The extent of the relationship between the results of the different pro- 
cedures can, perhaps, be more easily sensed from the scatterplots of the data 
in Figures lA through IC. Considering that approximately six months passed 
between the collection of the CRT and CRK data and the collection of the SRK 
data, and that the SRK data was rtot limited to the^opinions of principals but 
al50 /ncluded?' the opinions of school superintendents and ^issistant superinten- 
dents, it is reasonable to interpret the obtained correlation coefficients as 
conservative estiinates of the dtegree to which the results of the ,^fferent pro 
cedures are actually related. 
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TAI^LE 1 

Les By The Subset Ranking Procedure (SRK) , 
md Number of Administrators Ranking Each Goal 
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8R 4.222 9 

13F 4.375 16 

_2B 4.500 10 

37R 4.692 13 

27A -4.737 19 
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9R 4.933 15 

4nR 5 . 300 10_ 

4nA 5.333 9 

23A 5.556 9 
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871 5.600 10 

8t: 5.700 10 
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IC 5.800 5 
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41A 7.364 11 
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figure lA 

vattcqUot of results obtained by the Subset Ranking Procedure (SRK) 
v.ith results obtained by the Complete Ratinp Procedure fCRT)* 
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*Lach circle fo) represents a specific goal, H.q., goal 41B was ranked 
first by the CRT but third by the SRK. 



Figure IB 



Scatterplot of results obtained by the Subset Ranking Procedure (SRK) 
with the results obtained by the Complete Rating -Ranking Procedure (CRT) 
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Figure IC 



Scattcrplot of results obtained by the Complete Rating Procedure (CRT) 
with the results obtained by the Complete Rating-Ranking Procedure (OiK) 
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\^hen school administrators choose to undertake a Needs Assessment, how- 
ever, they are primarily concerned with identifying "high priority goals." 
For this reason, studies into the possibility o£ employing subset-type proce- 
dures ^s alternates or substitutes for "ccHiiplete" procedures must carefully 
examine the degree to \\iiich subset- type procedures identify the sayie high 
priority goals. 

Tables 3A through 3C present the eleven highest ranking goals identified 
by the SRK, CRT, and the CRK in the present study. The "connecting > lines" 
identify goals that were placed in the top eleven ranks by both of the pro- 
cedures COTipared. The connecting lines in Tables 3A and 3B indicate that the 
SRK was successful in identifying eight of the eleven top-ranked CPT and CRK 
goals. The crossed connecting lines, however, indicate that the eight goals 
were not ranked in the same order. 

This difference in goal priorities (i.e., as reflected in crossed c6n- 

nectir.g lines) is not as damaging to the argument for using the SRK procedure 

as an alternate or substitute for "complete" procedures as one might at^irst 

2 

believe. The crossed connecting lines in Table 3C suggest that two different 
goal prioritizing procedures will simply yield different results- -even if the 
same exact group of raters is involved in the test of this proportion. 

The importance of differences in priorities in top-ranked goals is 
further mitigated by the "reliability" of our instruments. Even if the pro- 
cedures under consideration (SRK, CRT, and the CRK) were (approximately) per- 
fectly reliable, with the standard deviations of goal ratings/rankings being 

3 

as large as they are, the Standard Error of Measurement for almost every sin- 
gle goal would be 'such, that there would still be a 95% chance that a goal's 

4 

true rank could actually lie within +2 ranks from its obtained rank . If the 
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reliability of any of the procedures was as low as r^^ = ,91, givim the mag- 
nitude of goal staridard deviations and the standard error of measurement, 
there would be a 95% chance that almost every goal's true rank could actually 
lie within a range of 2 to 10 ranks from its obtained rank. 

The degree of uncertainty within any obtained set of data is such that 
no matter which goal prioritizing procedure the administrator chooses to 
employ, he cannot with confidence make iinportant differential decisions solely 
on the basis of one goal's having a slightly higher mean rating/ranking than 
another goal, 

DISCUSSION 

The data indicate that the subdeck ranking procedure yields results 
that are essentially equivalent to *'cOTiplete procedure" results (this, even 
with the potentially biased nature of the sainples, the bias working against 
any hypothesis of equ^. valence)'. The equivalence of SRK and '^ccxnplete pro- 
cedure" results is, however, just one of several factors that needs to be 
examined before any decision to recommend the SRK as an alternate or sub- 
stitute can be made. Additional factors'^'include (1) the extent to vrfiich 
"subdeck procedures" address problems associated with the use of "complete 
procedures"; and (2) potential benefits and problems of "subset procedures." 

Problems reported (Hoepfner, 1971)^ by (scwne)principals during the na- 
tional field testing of the section of the Needs Assessment KIT that had to 
do with identifying goal priorities with the Collective Viewpoints Approach 
included the following: 

1. Too many goals 

2. Finding teacher time - 

3. Not enough cards 



4. 



Tallying time too consuming 



5. 



Parent understanding of directions 



6. Getting returns from parents 

7, Lack of cooperation 

8. Only one socioeconomic group represented 

9, Too few subjects 
10. Parent availability 

^ 11, Goals interpreted differently by, different groups 
It seems reasonable to speculate that employing a subdeck procedure will, at 
least in part, deal with problems 1 through 4, and possibly mitigate problem 
5. Subset* type procedures, however, do not seem to have any potential for 
addressing problems 6 through 11. 

In examining the possibility of using the SRK °(or other subdeck procedures) 
as an alternate or substitute for ccxnplete procedures, it is important to con- 
sider that the field test report on this section of the KIT showed that: 

1. "School principals felt that the system of rating goals was easy 
for themselves (89%), easy for teachers (93%), and to a lesser 
extent easy for the parents (70%)."^ 

2. "Ninety-five percent of respondents found that the instructions/ 

for the Collective Viewpoints Approach were understandable; 93 percent 
could follow. the procedure; and 85 percent found the computations 
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4. "Parent attitudes 5 as reported by principals, were 70 percent 
favorable and 20 percent mixed; again 10 percent of the princi- 

o 

pals offered no report on parent attitudes." 

5. "The overall reaction of the respondents to Booklet II (goal 

ratings) was 75 percent favorable, 11 percent mixed, and 4 percent 

0 

unfavorable; v^ile 6 percent had no ccsnment."" 

6. "Eighty-two percent of the respondents would recommend the pro- 
cedures in Booklet II (goal ratings) to other principals.""^^ 

Based on speculation, the most serious potential problem of the SRK or 
any other subdeck procedure, is that individual raters are likely to be dis- 
satisfied about determining the priorities within a set of goals that is not 
likely to contain all,^<)r perhaps any, of the goals they believe are most im- 
portant. 

CONCLUSION 

i ■ 

The subset ranking procedufe (SRK) yields results that ^re equivalent to 
those obtained with "complete" rating or ranking procedures. However, it seems 
inappropriate at this time to either recommend or reject tne use of the SRK or 
any other subdeck procedure as an alternate or substitute for "complete" 
rating/ranking procedures until the potential problem of rater dissatisfaction 
can be investigated. 
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Notes 



The reader should be aware that r and T have different underlying,, scales 
and are therefore not directly co^arable (Siegel, 1956, p. 220). r^ is 
derived from the formula for a Pearson Product -Manent Correlation; while 
T is derived from the formula for the binomial distribution. Hays (1963, 
p. 649) points out that while the Spearman Coefficient is meaningful, at 
least at an elementary level, only by analogy with the ordihn^^ correla- 
tion coefficient, the interpretation of an obtained value of . is straight 
forward. If a pair of objects (e.g., goals) is drawn at randcxn from among 
those ranked, the probability that these two objects show the same order 
in both rankings is (eig., if T = .65) .65 more than the probability il\?X 
they woiild show differmt order. In other words, from the evidence at 
hand, it is a considei^ly better bet that the two procedures will tend 
to order a rand(mL)^^^lected pair in the same way than in a different way. 

The reader should remember that both the CRT and CRK results were based 
on the judgments of the s ame 53 principals. And, . that the CRIt results . 
rest on the CRT results. 

Standard Error of Measurement = s^\J 1-r^^ See Magnusson (1966, p. 79-80). 
P = .95 

Hoepfner, 1971, p. 33 
op. cit . r p. 35 ^ 
op. cit. , p. 35 
op. cit . , p. 35 
op. cit. , p. 36 
op. cit. , p. 36 
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■ APPENDIX: 106 Goals of Fla.ientary Educat 

IA. Shyness -Boldness 

IB. Neuroticism- Adjustment 

IC. General Activity- Lethargy 
2A . Dependence - Independence 
2B. Hostility-Friendliness 

2C . Soci al izat ion - Rebel 1 iousness 

3A. School Orientation 

3B. Self-Estean 

4A. Need Achievement 

4B. Interest Areas 

r * 

5A. Appreciation of Arts and ^Crafts 

SB. Involvement in Arts and Crafts 

6A. Representational Skill in Arts and Crafts 

6B. Expressive Skill in Arts and Crafts 

7A. Arts and Crafts Ccmprehension 

7B. DeveloF?nental Understanding of Arts and Crafts 

8A. Classificatory Reasoning 

8B. Relational-Implicational Reasoning 

8C. Systematic Reasoning 

8D. Spacial Reasoning 

9A. Creative Flexibility 

9B. Creative Fluency 

lOA. Span and Serial Memory 

lOB. Meaningful Memory 

IOC. Spacial Memory 

llA. Reading Comprehension of a Foreign Language 
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. ^IB. Oral Comprehension of a Foreign Language 

lie. Speaking Fluency of a Foreign Language 

IID. Writing Fluency in a Foreign Language 

12A. Cultural Insight through a Foreign Language 

12B. Interest in and Application of a Foreign Language 

13A. Spelling 

13B. Punctuation 
■ 

13C. Capitalization 

13C. Grammar and Usage 

13E. Penmanship 

13F. Written Expressiwi 

13G. Independent Application of Writing Skills 

14A. Use of Data Sources as Reference F'liills 

14B. Sumnari zing Information fort Reference 

ISA. Comprehension of Nuntiers and Sets in'Mathematics 

15B. Comprehension of Positional Notation, in Mathoonatics 

15C. Comprehension' of Equations and Inequalities 

15D. Canprehension of Number Principles 

16A. Operations with Integers 

16B. Generations with Fractions 

16C. Operations with Decimals and Percents 

I7A. Mathematical Problem Solving ' 

17B. Independent Application of Mathaflatica^ Skills 

ISA. Geometric Facility 

IRR. Geonetric Vocabulary 

19A. Measurement Reading an J Making 

19B. Statistics 



20A. Music Appreciation 

20B. Music Interest and Iinjo>'inent 

21A. Singing 

21B. Musical Instrument Playing 

21C. Dance (Rhythmic Response) 

22A. Aural Identification of Music 

22B. Niusiv Knowledge 

23A. Practicing Health and Safety Principles 

Z3B. Understanding Health and Safety Principles 

23C. Sex Education 

24A. Muscle Control (Physical Education) 

24B. Physical Development and Well-Being (Physical Education) 

2-5A. Group Activity - Sportsmanship , ^. 

25B.^ Interest and Independent Participation in Sports § Games'^ 

26A. Understanding Riiles ^ Strategies of Sports § Games 

26B. Knowledge of Physical Education Apparatus and Equipment 

27A. Listening Reaction and Response to Reading 

27B, Speaking * } 

28A. Phonetic Recognition^ 

28B. Structural Recognition 

29A, Oral Reading 

29B, Silent Reading Efficiency 

30A. Recognition of Word Meanings 

30B. Understanding Ideational Complexes 

30C. Remembering Information Read c ^ 

31A. Inference Making fran Reading Selections 

siB. , Recognition of Literary Devices 
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31C, Critical Reading 

32A, Attitude toward Reading 

32B. Attitude and Behavior Modification from Reading 

32C. Familiarity with Standard Children's Literature 

33. Religious Knowledge 

34. Religious Belief 

35A. Observation and Description in Science 

35B. Use o± Numbers and Measures in Science 

35C. Classification and Generalization in Science 

35D. Hypothesis Formation in Science 

35E. Operational Definition^ in Science 

35F. ExperimenLation in Science 

35G. Fonnation of Generalized Conclusions in Science 

36A. Knowledge of Scientific Facts and Teiminology 

36B. The Nature and Purpose o£ Science 

37A. Science Interest and Appreciation 

37B. Application of Scientific Methods to Life 

38A.- Knowledge of History 

^ 38B. Knowledge of Governments 

39A. Knowledge of Physical Geography 

39B. Knowledge of Socio-Econcmic Geography 

40A. Cultural Knowledge 

40B. Social Organij^ation Knowledge 

41A. Research Skills in Social Sciences 

41B. Citizenship 

41C. Interest in Social Studies 
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